The present invention relates generally to electronic whiteboard systems and, more particularly, to projector and camera arrangements and optical markers for use in such electronic whiteboard systems.
An electronic whiteboard system is a processor-based computing system used to input and output information associated with a software application running on the system. Typically, in accordance with such a system, one or more users xe2x80x9cwritexe2x80x9d on the whiteboard using an electronic writing instrument or marker, such as a lightpen. The lightpen permits the user to write with xe2x80x9celectronic ink.xe2x80x9d Electronic ink is the term given to writing that is electronically captured from and/or electronically projected on the whiteboard without using physical ink. A user""s writing, as well as any other desired information, is displayed on the whiteboard which is viewable by the one or more users. The data entered on the whiteboard may then be stored for subsequent use by the application being run on the system. Examples of such whiteboard systems are: Ideaboard by 3M Inc. (http://www.3m.com/vsd/ams/11_whtbd.html); e-Bearn by Electronics for Imaging, Inc. (http://www.e-beam.com/index_flash.html); SoftBoard by Microfield Graphics, Inc. (http://www.softboard.com/); SMART Board (htttp://www.smartboard. co.uk/product/index.html); Mimio by Virtual Ink Inc. (http://www.virtual-ink.com/ns. shtml); and Liveboard, The Office of the Future: Xerox PARC, Wendy Taylor, PC Computing, pp. 192, January 1995.
Electronic whiteboard systems may be collaborative. A collaborative whiteboard system is a distributed computing system which includes two or more individual electronic whiteboard systems, as mentioned above, in communication with each other while running a collaborative application. While the individual systems, and thus their respective users, are remote from one another, a user at a first location is able to view information written by a user at a second location. This way, the remote users may interact as if they are in the same location. Examples of such whiteboard systems are: Netmeeting by Microsoft, Inc.; and Sametime by Lotus, Inc. (an IBM company).
Whether a single stand-alone system or a collaborative system, a typical electronic front-projection whiteboard system, as illustrated in FIG. 1A, is comprised of a whiteboard screen 2, a writing stylus or lightpen 4, a fixed-position projector 6, a fixed-position camera 8 and a processing system 10. In such a system, the function of projecting images representing a user""s writing on the whiteboard screen 2, in accordance with the lightpen 4, is performed by the fixed-position projector 6. As shown in FIG. 1A, the projector 6 has its own imaging optics 7 associated therewith. The fixed-position camera 8, aimed at the writing substrate 2 and the lightpen 4, captures an image of the whiteboard and the light emitted by a lamp associated with the lightpen. Like the projector 6, the camera 8 has its own imaging optics 9 associated therewith. Suitable optical and electronic filtering assure that only the lightpen is sensed among possible background clutter and distractions such as other bright objects. As is known, the presence and location of the lamp of the lightpen in the field of view of the camera may be estimated by various signal processing techniques. Several technologies exist that address the problem of capturing the location of a stylus on a whiteboard. For example, one such technique is used in Xerox""s LiveBoard as mentioned in S. Elrod et al., xe2x80x9cLiveboard: A Large Interactive Display Supporting Group Meetings, Presentations and Remote Collaboration,xe2x80x9d CHI ""92 May 3-7, 1992, pp. 599-607.
The images projected by the camera on the screen, representing the user""s writing strokes, are derived from a display screen buffer. The contents of the display screen buffer depend on optical screen marking events such as those generated by the lightpen. The visual effect that the user""s strokes are physically being written on the whiteboard is achieved by the camera projecting the image of the optical marker or lightpen path onto the board.
As is known, the processing system 10 includes processor and memory resources for coordinating the functions performed by the whiteboard screen 2, the lightpen 4, the projector 6 and the camera 8. Accordingly, the system must accurately sense the location of the lightpen on the board and then project its writing actions onto the board. One method for accomplishing these tasks is as follows. The camera and its imaging optics are aimed at the board in order to capture the optical emission from the lightpen. The captured position of the light must then be transformed such that the projected writing trace generated by the projector appears at the tip of the lightpen as it writes. The transformation used to achieve this goal depends on many factors such as the settings and location of the imaging optics of the projector, and the settings and location of the imaging optics of the camera. However, determining the transformation can be a problem.
Such problem associated with determining the transformation can be generically described as follows. A processor produces an image that is being projected onto a physical surface. Find the spatial relationship of a visual marker, which is not necessarily controlled by the processor, relative to the projected image, for fixed optics and settings of the projector. Consider the case in which the marker is observed by a camera. Once this relationship is found, the processor that drives the projector uses the information about the marker to produce visual effects such as mimicking the action of the lightpen on the board.
In accordance with existing whiteboard projector/camera arrangements, the problem is typically solved by employing a calibration procedure, wherein the camera and its optics are calibrated with the projector and its optics so as to determine the proper transformation. Unfortunately, such a calibration procedure is typically disruptive, time consuming and usually ignores lens radial distortion and other hard-to-correct optical lens aberrations.
The same problems also exists in rear-projection whiteboard system. A conventional rear-projection whiteboard system is illustrated in FIG. 1B. The system is comprised of a whiteboard screen 2xe2x80x2, an enclosure 3, a writing stylus or lightpen 4xe2x80x2, a reflecting mirror 5, a fixed-position projector 6xe2x80x2, a fixed-position camera 8xe2x80x2 and a processing system 10xe2x80x2. The components and their functions in the rear-projection system in FIG. 1B are essentially the same as those in the front-projection system in FIG. 1A, as described above, with the following exceptions. In the front-projection system, the user is on the same side of the whiteboard screen as the projector, while in the rear-projection system, the user and the projector are on opposite sides of the screen. Also, the screen 2xe2x80x2 in the rear-projection system is typically translucent so that the lightpen 4xe2x80x2 can be tracked by the camera 8xe2x80x2, via the reflecting mirror 5, and so that the user on one side of the screen can view the images projected on the other side of the screen by the projector 6xe2x80x2, via the mirror 5. Like the conventional front-projection system, the projector and camera of the rear-projection system each have their own separate imaging optics 7xe2x80x2 and 9xe2x80x2, respectively. Thus, a similar calibration procedure must be performed to determine the appropriate transformation relationship which, as mentioned above, presents many operational drawbacks.
Thus, it would be highly desirable to solve the problem associated with determining the above-described transformation relationship associated with the projector and camera of an electronic whiteboard system such that the disadvantages associated with the use of a calibration procedure could be avoided.
The present invention provides projector and camera arrangements for use in such electronic whiteboard systems. Specifically, the present invention provides projector and camera arrangements wherein the projector and camera share the same imaging optics. By sharing the same projection and camera optics, the distortions that affect the projection system are the same as those of the camera system. Thus, the calibration step required in conventional whiteboard systems where the projector and camera are separate, i.e., each having their own distinct optics and settings, is no longer needed. Further, the arrangements provided in accordance with the invention are self-aligning, even when lens distortions are large and even in the presence of strong perspective effects. The shared optics projector and camera arrangements of the invention also provide for dynamic zooming. It is to be appreciated that the invention applies to both front-projection and rear-projection whiteboard systems, as well as other possible projection arrangements.
Thus, in one aspect of the present invention, an image capture and projection apparatus for use in an electronic whiteboard system, comprises: (i) an image capture device, the image capture device tracking a position of at least one stylus associated with the electronic whiteboard system used to enter data in accordance with a surface associated with the electronic whiteboard system; and (ii) an image projection device, the image projection device projecting an image which is viewable on the surface, in proximity of the position of the stylus, and representative of the data entered in accordance with the stylus, wherein the image capture device and the image projection device share at least one imaging lens for capturing and projecting one or more images. It is to be appreciated that the image capture device and the image projection device are preferably physically integrated with one another to form an integrated image capture/projection device.
The present invention also provides active and passive optical markers or lightpens for use in electronic whiteboard systems. Such inventive lightpens preferably provide the following advantages over existing lightpens: (i) they accommodate large whiteboards allowing several people to gather around and contribute; (ii) they are cordless; (iii) they allow writing on non-planar surfaces; (iv) they are robust; and (v) they are extensible to three-dimensional space.
In one aspect of an optical marker of the present invention, an active optical marker device for use in accordance with an electronic whiteboard system, wherein the electronic whiteboard system includes a data entry surface, an image capture device for tracking a position of the optical marker device while the optical marker device is used to enter data in accordance with the data entry surface, and an image projection device for projecting an image which is viewable on the surface, in proximity of the position of the optical marker device, and representative of the data entered in accordance with the optical marker device, comprises: (i) an infrared light-emitting source; and (ii) a switch, operatively connected to the infrared light source, and operative to turn on the infrared light source when the optical marker device contacts the surface and turn off the infrared light source when the optical marker device does not contact the surface, such that the image capture device can capture the infrared light emitted by the optical marker device and the image projection device can project the representative image on the surface.
In one embodiment, the infrared light-emitting source is directed toward the image capture device. In another embodiment, the surface is a reflective surface and the infrared light-emitting source is directed toward the surface. The active optical marker device may also have a low-friction nib connected to the switch, wherein the nib contacts the surface when data is being entered.
In another aspect of an optical marker of the present invention, a passive optical marker system for use in accordance with an electronic whiteboard system, wherein the electronic whiteboard system includes a data entry surface, an image capture device and an image projection device, comprises: (i) an infrared-emitting light source for illuminating the data entry surface; and (ii) an infrared light reflector having a low-friction glide and a non-isotropic surface, the reflector being worn by a user such that the low-friction glide comes into contact with the surface when the user enters data and, in accordance with such data entry, infrared light emitted by the source is reflected from the reflector to the image capture device, such that the image capture device can capture the reflected infrared light and track a position of the reflector, and such that the image projection device can project an image which is viewable on the surface, in proximity of the position of the reflector, and representative of the data entered in accordance with the reflector on the surface.